1. Field of the Invention
The present invention relates to an image processing apparatus for processing image data obtained by a CCD (charge-coupled device) area sensor having a primary-color filter so as to obtain a high quality image. Such an image processing apparatus is mounted on a digital camera (e.g., electronic still camera) or the like.
2. Description of the Related Art
A conventional image processing apparatus for use in a digital camera, or the like, performs a series of various image processing steps, e.g., interpolation for each color component, contour emphasizing processing, etc., on image data obtained by a CCD area sensor having a primary-color filter, so as to obtain a high quality image. Hereinafter, conventional image processing apparatuses are described with reference to FIGS. 13 through 21.
FIG. 13 is a block diagram showing a first exemplary structure of a conventional image processing apparatus. In FIG. 13, the image processing apparatus 100 includes an optical low-pass filter 101, a primary-color CCD area sensor 102, a RGB interpolation section 103, a white balance adjustment section 104, a gamma correction section 105, and a sharpening processing section (contour emphasizing section) 106.
The low-pass filter 101 removes frequency components whose frequency is equal to or higher than a ½ of sampling frequency fs. According to the sampling theorem, when an image is converted into image data, frequency components whose frequency is equal to or higher than a ½ of sampling frequency fs are converted into aliasing noise. In order to reduce such aliasing noise, the optical low-pass filter 101 is provided before the CCD area sensor 102.
The primary-color CCD area sensor 102 is formed by a plurality of light-receiving elements arranged in a matrix. The primary-color CCD area sensor 102 includes a primary-color (R, G, B) filter on a light-receiving face thereof where each of color areas corresponds to one pixel. The color filter is formed based on the Bayer array (FIG. 14) which is a type of RGB arrangements.
The RGB interpolation section 103 (which will be described later in detail) can receive, from the CCD area sensor 102, only one type of color component for each pixel among R, G, B colors of the color filter of the sensor 102. In the RGB interpolation section 103, each of the remaining two types of color components in each pixel is calculated from color components of the same type in neighboring pixels, whereby all of the three color components are obtained in each pixel.
The white balance adjustment section 104 adjusts the white balance with respect to the R-, G-, and B-components in each pixel which are obtained by the RGB interpolation section 103 according to the color temperature of light so as to correct the color of an image.
The gamma correction section 105 processes the R-, G-, and B-components obtained after the white balance has been adjusted such that the R-, G-, and B-components conform to properties of a display or printer which outputs an image.
The sharpening processing section 106 performs contour emphasizing processing (sharpening processing) in order to obtain a sharp image. The sharpening process compensates for high-range luminance components which have been removed by the low-pass filter 101 by filtering processing which emphasizes high-range luminance components. That is, in the case where the low-pass filter 101 is provided, CCD image data generated by the primary-color CCD area sensor 102 has a reduced high-range luminance component. Furthermore, for example, since an unobtained color component in each pixel is compensated for by interpolation, e.g., by calculating an average value from color components of the same type in neighboring pixels, an image obtained by such interpolation has a further reduced high-range luminance component. Thus, in order to obtain a sharp image, it is necessary to compensate for a lost high-range luminance component by performing a filtering process which emphasizes high-range luminance components.
Interpolation by the RGB interpolation section 103 is now described in detail.
When the CCD area sensor 102 is a single-plate area sensor, in the RGB interpolation section 103, interpolation is performed for the missing two types of color components such that all of R(red)-, G(green)-, and B(blue)-color components are complete in each pixel. Based on all of R-, G-, and B- color components including color components obtained by interpolation, a color image is created.
When the CCD area sensor 102 is a three-plate area sensor where three area sensor elements are provided with full-size R-, G-, and B-color filters, respectively, the RGB interpolation section 103 receives from the CCD area sensor 102 all of R-, G-, and B-color components for each pixel. Thus, it is not necessary to perform interpolation. However, when the CCD area sensor 102 is a single-plate area sensor having, on its light-receiving element, a single color filter on which R-, G-, and B-filter areas are arranged in a predetermined pattern (e.g., Bayer array shown in FIG. 14), it is necessary to perform interpolation for missing color components in each pixel. Therefore, when employing a single-plate CCD camera incorporating a single-plate CCD area sensor, missing color components in each pixel which cannot be obtained by a color filter are created by various methods.
U.S. Pat. Nos. 4,605,956, 4,642,678, and 4,630,307, and the document by James E. Adams, Jr., “Interactions between color plane interpolation and other image processing functions in electronic photography”, suggest various interpolation methods for the purpose of creating an image with no jaggy or zip noise. “Jaggy” means a step-shaped noise which emerges along a contour of an image, which is a type of noise among various types of noise.
In U.S. Pat. No. 4,605,956, interpolation is performed by using the following expressions (1) through (4) on missing color components in a pixel arrangement shown in FIG. 15 (bilinear interpolation method). Interpolation for G- and B-components at each pixel position in FIGS. 14 and 15 is shown below:G5=(G2+G4+G6+G8)/4  (1)B5=(B1+B3+B7+B9)/4  (2)B2=(B1+B3)/2  (3)B4=(B1+B7)/2  (4)
In the Bayer array shown in FIG. 15, R-, G-, and B-colors are arranged according to a certain pattern. For example, at a central pixel position, R5, color components G5 and B5 are missing in a corresponding color filter. The color components G5 and B5 are obtained from expressions (1) and (2).
Furthermore, B2 is obtained from expression (3), and B4 is obtained from expression (4). Interpolation for each of B6 and B8 is performed by obtaining an average value of B-component values in vertically or horizontally neighboring pixels based on an expression similar to expression (3) or (4). This is the same for R2, R4, R6, and R8.
Furthermore, G1 is obtained from G-component information in neighboring pixels around pixel position “B1” by an expression similar to expression (1). This is the same for G3, G7, and G9. R1 is obtained from R-component information in neighboring pixels around pixel position “B1” by an expression similar to expression (2). This is the same for R3, R7, and R9.
Next, an image processing method disclosed in Japanese Laid-Open Publication No. 10-164371 is described with reference to FIG. 16.
FIG. 16 is a block diagram showing a second exemplary structure of a conventional image processing apparatus. As shown in FIG. 16, the image processing apparatus 200 includes a primary-color CCD area sensor 201, a RGB interpolation section 202, and a contour emphasizing sections 203 provided for respective ones of R-, G-, and B-components.
The RGB interpolation section 202 first interpolates a G-component by using expression (5) which represents median processing for G-color components obtained by the filter arrangement shown in FIG. 15. Then, only at pixel positions, “R” and “B”, in FIG. 14, a (R-G) component and a (B-G) component are created. In the last, the created (R-G) and (B-G) components are interpolated by bilinear interpolation, and G-components are added to the interpolated (R-G) and (B-G) components, whereby R- and B-components are obtained.G5=(G2+G4+G6+G8−Min−Max)/2  (5)                Min=Min(G2,G4,G6,G8)        Max=Max(G2,G4,G6,G8)        
How to obtain B4 at pixel position “G4” is now described in detail with reference to FIG. 15. In the first step, G1, G3, G5, G7, and G9 are interpolated by the median processing process using expression (5). Then, (B1−G1) and (B7−G7) which are (B-G) components at pixel positions “B1” and “B7” are created. (B4−G4) which is a (B-G) component at pixel positions “G4” is represented as follows:B4−G4=(½){(B1−G1)+(B7−G7)}From this expression, B4 can be represented as follows:B4=(½){(B1−G1)+(B7−G7)}+G4
The contour emphasizing sections 203 employs a two-dimensional second-derivative filter shown in FIG. 17A for each of R-, G-, and B-components. Each box of the two-dimensional second-derivative filter shown in FIG. 17A corresponds to one pixel, and the number shown in each box represents a weight. The weight of each box area is set such that the total weight in the two-dimensional second-derivative filter is zero.
Next, an interpolation method disclosed in Japanese Laid-Open Publication No. 11-18047 is described with reference to FIG. 18.
FIG. 18 is a block diagram showing a third exemplary structure of a conventional image processing apparatus. In FIG. 18, the image processing apparatus 300 includes a primary-color CCD area sensor 301, RGB interpolation sections 302, a middle-range component emphasizing section 303, a high-range component emphasizing section 304, a white balance adjustment section 305, and a gamma correction section 306. The primary-color CCD area sensor 301 includes a color filter of Bayer array.
The RGB interpolation sections 302 perform interpolation on each of R-, G-, and B-components. When a G-component is interpolated by using a median method of expression (5), and R- and B-components are interpolated by using a bilinear method of expressions (2) through (4), interpolation is achieved with high image quality.
The middle-range component emphasizing section 303 is formed by a middle-range component extraction section 303a and adders 303b. In the middle-range component emphasizing section 303, the middle-range component extraction section 303a extracts a middle-range component from a G-component interpolated by the G-interpolation section of the RGB interpolation sections 302. The extracted middle-range component is synthesized with each of R-, G-, and B-components by the adders 303b, whereby the middle-range component emphasizing processing is achieved. In this middle-range component emphasizing processing, the difference between a G-component and a low-pass filtered G-component is added as a compensation component to the G-component. In other words, a high-range component of the G-component is removed by a low-pass filter (not shown) to obtain a GBAR-component, and a compensation component which is the difference between the G-component and the GBAR-component (G-GBAR) is added to the G-component, whereby a middle-range component emphasizing processing is achieved.
The high-range component emphasizing section 304 is formed by a high-range component extraction section 304a and adders 304b. In the high-range component emphasizing section 304, the high-range component extraction section 304a extracts a high-range component from a G-component interpolated by the G-interpolation section of the RGB interpolation sections 302. The extracted high-range component is synthesized with each of R-, G-, and B-components by the adders 304b, whereby high-range component emphasizing processing is achieved. This high-range component emphasizing process employs the two-dimensional second-derivative filter shown in FIG. 17B. In the filter shown in FIG. 17B, in order to obtain pixel data of a pixel corresponding to a central box “4”, the weight of the filter is multiplied by data from each pixel, and the products for all of the pixels are added up. Thus, when a result of this calculation does not change, the sum of the products results in zero. When a result of this calculation changes much, the sum becomes a large value, whereby a high-range component is emphasized.
Now, consider a case where each of R-, G-, and B-components obtained through a Bayer array filter shown in FIG. 14 is sampled at a sampling frequency fs=1/Δx=1/Δy. A sampling frequency distribution range obtained in this case is shown in FIG. 19. Herein, “Δx” denotes the width of a pixel (pixel pitch) in a horizontal direction (x-direction), and “Δy” denotes the width of a pixel (pixel pitch) in a vertical direction (y-direction).
According to the sampling theorem, a highest restorable frequency within a spatial frequency band of an original image is represented by a solid-line lozenge which is indicated by an arrow “G-component” in FIG. 19. Furthermore, R- and B-components are included in a two-dot chain line square. Thus, a frequency band which can be accurately converted into R-, G-, and B-components of image data is an area within the square defined by the two-dot chain line. As seen from FIG. 19, the accurately-restorable, highest frequency of the original image is a half of the sampling frequency fs(=1/Δx=1/Δy).
Therefore, a frequency component which is higher than the highest restorable frequency fs/2 emerges as a noise in the image data.
In general, for the purpose of avoiding such a problem, an optical low-pass filter (anti-aliasing filter) is provided to a CCD area sensor. This low-pass filter removes frequency components of image data which are higher than the highest restorable frequency fs/2, but undesirably attenuates some of frequency components which are lower than the highest restorable frequency fs/2 because the low-pass filter is not ideal. In FIG. 20, graph a shows an ideal low-pass filter frequency characteristic (i.e., only frequency components which are higher than the highest restorable frequency fs/2 are removed). However, in an actual case, the low-pass filter frequency characteristic results in a curve of graph b. Furthermore, graph a shows an ideal frequency characteristic of a compensation filter which is used to compensate for the characteristic of graph b so as to obtain a characteristic similar to the ideal characteristic of graph a. According to the present invention, as described later in detail, a middle-range luminance component and a high-range luminance component are synthetically added to a newly-extracted, middle-high range luminance component according to a predetermined synthetic ratio, whereby a low-pass filtered component is compensated for by using a compensation filter having a frequency characteristic similar to graph c.
Furthermore, when interpolation is performed on components of an image which have been obtained through a Bayer array filter shown in FIG. 14 so as to compensate for the missing two types of color components in each pixel, high-range luminance components are attenuated. Accordingly, it is indispensable for generating a sharp image to compensate for attenuated high-range components. In general, such a compensation is achieved by separately performing a compensation for middle-range luminance components with a compensation filter having a frequency characteristic of graph d and a compensation for high-range luminance components with a compensation filter having a frequency characteristic of graph c. In FIG. 21, graph b shows a synthetic frequency characteristic obtained after low-pass filtering (with an anti-aliasing filter) and interpolation processing, and graph a shows an ideal frequency characteristic obtained after an entire process in an image processing system including compensation processing.
In the above-described conventional art, attenuated middle- and high-range components among low-pass filtered components are compensated for by compensation filters. The middle-range components are compensated for by a compensation filter having a characteristic represented by graph d of FIG. 21, and the high-range components are compensated for by a compensation filter having a characteristic represented by graph a of FIG. 21. When graph d exhibits the maximum amplitude, the angular frequency ω is π/2 (equivalent to fs/4). When graph a exhibits the maximum amplitude, the angular frequency ω is π (equivalent to fs/2).
However, in contour emphasizing processing which is performed for obtaining a sharp image, when high-range components of an image are emphasized by compensating for high-range components, all components having an angular frequency ω higher than fs/2 are converted into noise. Thus, as the resolution of the image is increased, noise and jaggy are observed more prominently.
For example, in the method described in connection with the first exemplary conventional structure shown in FIG. 13, a G-component which largely contributes to the luminance of an image is interpolated according to above expression (1), and image data obtained after the interpolation is subjected to contour emphasizing processing, and as a result, jaggy emerges in an edge portion of the image. Examples of proposed method for solving such a problem include a color-smoothing interpolation method (U.S. Pat. No. 4,642,678), a pattern-recognition interpolation method (U.S. Pat. No. 4,630,307), an adaptive interpolation method (document by James E. Adams, Jr.). These methods are different interpolation methods, but all performed in a structure similar to the image processing apparatus 100 shown in FIG. 13. That is, a high-range component which is indispensable for obtaining a sharp image is generated by contour emphasizing processing in the sharpening processing section 106 which resides at the end of the entire process. These interpolation methods each have some advantages in reducing false color and jaggy. However, these interpolation methods do not have means for processing a high-range component except for high-range emphasizing filtering, and thus, noise occurs in the high-range emphasizing filtering.
Thus, in the methods described in connection with the above first through third exemplary conventional structures (FIGS. 13, 16, and 18), noise is inevitably emphasized when a high-range component is compensated for by sharpening processing. That is, during interpolation of high-range components, frequency components whose frequency is equal to or higher than ½ of a sampling frequency fs are also emphasized, and the emphasized components whose frequency is equal to or higher than a ½ of sampling frequency fs are observed as noise and/or jaggy.